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ABSTRACT 

The components of complex domains have dynamic, 
interdependent relationships that are often unstable. A useful method for 
representing relationships among elements in complex systems is causal 
influence diagramming, often described as concept mapping. This study employs 
the use of software applications, traditional assessment, and case studies to 
support systems analysis of learning theories through the development of 
causal influence diagrams (CID) and seeks to determine the extent to which 
causal influence diagramming affects performance on traditional assessment 
measures . A second goal of this study is to determine whether causal 
influence diagramming of a case study affects student performance on case 
study analysis. A partnership was formed between the faculty members 
responsible for teaching Educational Psychology courses in the elementary and 
secondary undergraduate programs. The faculty worked together to select 
appropriate software, identify important systems covered in the course, and 
establish a unified curriculum for teaching the systems targeted for the 
study. Students in undergraduate Educational Psychology courses were taught 
to use Inspiration software to express their understanding of causal 
relationships within learning theory and classroom systems. (Contains 13 
references, 7 figures, and 3 tables.) (AEF) 
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Systems Analysis of Learning Theory Through Causal Influence Diagrams 
Theoretical Framework 

The components of complex domains have dynamic, interdependent relationships 
that are often unstable. A useful method for representing relationships among elements in 
complex systems is causal influence diagramming, often described as concept mapping. 
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According to Jonassen, the act of concept mapping enhances the interdependence of 
declarative and procedural knowledge to produce structural knowledge, knowledge that 
provides insight into why things are the way they are (Jonassen, 1996). Theories of 
learning are complex systems that all teachers must be prepared to navigate in order to 
design instruction that is meaningful and appropriate. Educational psychology course are 
often the first place students are introduced to these complex systems. Modem 
technologies have increased awareness and understanding of how humans learn while 
providing new and efficient tools for expressing relationships within these complex 
systems. Providing a variety of methods for students to communicate their understanding 
of complex systems is an important goal for educators teaching courses on learning 
theories and developmental psychology. 

Advances in technology have stimulated interest in providing methods that 
support learning about complex domains (Spector and Davidsen, 1997), but there is no 
evidence that causal influence diagramming improves ability to analyze complex 
domains, although measurement instruments are currently under development. Spector 
suggests that tools borrowed from the realm of system dynamicists be employed to assess 
understanding of complex and dynamic systems (Spector, 2000). Dabbagh discusses the 
benefits of two concept mapping software tools that support the representation of 
complex systems and details the successes of previous studies on concept mapping and 
academic achievement (Dabbagh, 2001). Instructors and instructional designers now 
possess the resources to facilitate representation of complex systems through the creation 
of causal influence diagrams. Additionally, combining causal influence diagramming 
with a case-based approach to the study of learning theory provides students with a 
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complex system to analyze and a method for representing their understanding of the 
relationships among components within the system. 

This approach uses two pedagogical techniques that have been proven effective. 
Presenting a complex domain in the form of a case study allows prospective teachers to 
experience classroom life while encouraging detection of specific issues and problems 
within a safe, yet complex authentic context (Harrington, 1996). Contemporary software 
applications provide tools that support learning and analysis of complex domains 
(Spector & Davidsen, 1997), and classroom cases are easily obtained. This study employs 
the use of software applications, traditional assessment, and case studies to support 
systems analysis of learning theories through the development of causal influence 
diagrams (CID) and seeks to determine the extent to which causal influence diagramming 
affects performance on traditional assessment measures. A second goal of this study is to 
determine whether causal influence diagramming of a case study affects student 
performance on case study analysis. 

Methodology 

A partnership was formed between the faculty members responsible for teaching 
Educational Psychology courses in the elementary and secondary undergraduate 
programs. The faculty worked together to select appropriate software, identify important 
systems covered in the course, and establish a unified curriculum for teaching the systems 
targeted for the study. Students in undergraduate Educational Psychology courses were 
taught to use Inspiration software to express their understanding of causal relationships 
within learning theory and classroom systems. The software application facilitates the use 
of directional arrows, text fields, and graphics to indicate relationships between 
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components within a complex system. Enough licenses of the software permitted 
installation of the application on each student’s laptop for the semester, and the 
instructors taught the students to use the program during one 75-minute class period. 

Present Study 

The research design for the present study was refined after reflecting on the results of 
a pilot study. In the present study, students completed two descriptive causal influence 
diagrams and one analytical causal influence diagram. One descriptive diagram required 
students to demonstrate their awareness of the relationships between components of 
Piaget’s theory and the other descriptive diagram required students to demonstrate their 
awareness of the relationships between major components of the Individuals with 
Disabilities Education Act (IDEA). These tasks were considered “descriptive” because 
the causal influence diagrams identified the complexity of the relationships without 
critical analysis of how the various components relate to classroom practice. Students 
completed these diagrams outside of class and submitted them on the day the class 
focused on the respective topics. 

The analytical causal influence diagram required the students to determine 
motivational issues for a student case. This diagram was considered analytical because 
students were required to analyze the components of motivation represented in the case 
and show the relationship between the components and teaching strategies within a 
dynamic system. The instructor led students in an analysis and discussion of a 
motivational case prior to the deadline of the analytical diagram. Upon completion of 
class coverage of how motivation influences engagement and performance, students were 
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presented with a narrative that depicted a typical classroom motivation case about a 
fictitious student named Wayne (See Table 1). 

TABLE 1 . Causal Influence Diagram Assignments and Assessment Criteria 



Descriptive Causal Influence Diagrams 



Piaget (15) 



Design a causal influence diagram that reveals the relationships between components in Piaget’s theories. 
Criteria: Stages with minimum of 2 related concepts, adaptation, organization 



IDEA (15) 



Design a causal influence diagram that reveals the relationships between components of the IDEA. 

Criteria: Include components of IDEA from evaluation to due process and detail relationships between each 
component 



Analytical Causal Influence Diagrams 



Guided practice case Study — from Woolfolk, Anita. (2001) Educational psychology, S"' edition. Allyn and Bacon. 

Defensive Diana doesn’t have her lab manual-again. You tell her that she may share with another student. Then Diana pretends to be 
working, but she spends most of her time making fun of the assignment or trying to get answers form other students when your back is 
turned. She wants everyone to know that she “isn’t really trying.” That way, if her grades are low she has an excuse. She is afraid to 
try because if she makes an effort and fails, she fears that everyone will know she is “dumb.” 

CID assignment-Wayne’s case (15) 

You begin class. . . . “Get out your research notes, your prewriting notes, and the current draft of your persuasive essay. We’re going 
to continue writing and peer-editing body paragraphs today.” The moaning begins and several students have questions about your 
expectations. A few students pull out their notes and get started, but Wayne slumps in his chair and tries to become invisible. Wayne 
doesn’t have his research notes, but he does have a draft. He flips his papers and pretends to be editing his draft, making minor 
revisions and crossing out words and phrases. Wayne is unable to move forward with his essay because he has once again forgotten to 
bring important resources to class. Wayne always comes through in the end with a C or D quality product. Wayne doesn’t keep track 
of his grades or his work. Instead, he relies on his “sense” of his performance and occasionally checks his current grade average, just 
to get his bearings. It appears that Wayne is sabotaging his chances for good grades. His grades are somewhat better in chemistry and 
algebra, but it’s not like Wayne to try to impress anyone with his academic performance. 

Criteria: Design a CID that reveals the relationship between Wayne’s motivational needs and teaching strategies that might help 
Wayne improve his academic performance. Include: type of learner, locus of control, goals, self schemas (motivational style, self- 
efficacy, view of ability) 

Formal assessment-Bridgett’s case (5) 

Bridgett is usually the first student in the door for your World History class. She is an excellent student and is well liked by her peers. 
Helpful, prepared, and insightful, Bridgett possesses the work habits of an AP student. You’ve spoken to her about moving into a 
more advanced section of World History, but she refuses and says she doesn’t want to lose her “A”. 

Criteria: Analyze Bridgett’s motivational tendencies based on the concepts we discussed in class. You may represent your analysis 
through narrative, listing, or diagramming. Include: type of learner, locus of control, goals, self schemas (motivational style, self- 
efficacy, view of ability) 



Students were asked to create a list of influential components in the motivation case and 
to identify the influence of each component on other components (Figure 1). Students 
also created a causal influence diagram that depicted their analysis of the relationships 
between components of the motivation system depicted in the case (Figure 2). All causal 
influence diagrams were evaluated based on criteria established by experts in educational 
psychology. Table 1 shows the three causal influence diagram topics, the case used 
during the class on motivation, and the criteria used to evaluate the causal influence 
diagrams. 
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Figure 1. Chart outlining components and relationships for the analytical CID 
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In an effort to determine the extent to which causal influence diagramming affects 
performance on traditional assessment measures, multiple-choice questions not related to 
the causal influence diagramming activities were identified for two tests. Multiple-choice 
questions on subjects included in the causal influence diagramming activities were 
identified and labeled as lower and higher level cognitive processing based on Bloom’s 
taxonomy (Bloom, 1956). Multiple choice test items were also identified for the causal 
influence diagramming topics of Piaget, IDEA, and theories of motivation. 



Figure 2, Causal Influence Diagram Analyzing Motivation Theory in a Case Study 
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Upon conclusion of the final causal influence diagram, students were asked to 
respond anonymously to questions about causal influence diagramming and their 
perceptions of the effects on test performance. Data used to determine the extent to which 
causal influence diagramming influenced the ability to analyze complex systems came 
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from causal influence diagrams created by the students, performance on multiple-choice 
test items, and anonymous student responses to open-ended questions. 

Results 

High performance on causal influence diagramming and accuracy on multiple- 
TABLE 2, Descriptive Statistics for Performance on Causal Influence Diagrams 



Piaget 


IDEA 


Motivation 


Mean 


12.6 


11.4 


11.8 


Median 


12 


11 


13 


Mode 


12 


11 


14 


Std. Dev. 


1.9 


2.1 


3.6 



choice test items were common for students in this study. The causal influence diagram 
on Piaget was used to assess student knowledge of Piaget’s theories as well as student 
ability to manipulate the software. It was evident by the quality of the products that the 
students had no difficulties manipulating the software. Student performance on causal 
influence diagrams was measured by comparing performance to expectations established 
by experts (see Table 1). Each causal influence diagram was worth fifteen points and full 
credit was earned if specific criteria were represented accurately on the diagram. Students 
were expected to demonstrate awareness of the relationships between components by 
using directional arrows labeled with text descriptors of the relationship. Mean, median, 
mode, and standard deviation were determined for performance on each diagram (see 
Table 2). The difference between the means of the diagrams is minimal and the standard 
deviation is the greatest on the analytical diagram. The large deviation and the high mode 
on the analytical causal influence diagram indicate that a few members of the group had 
poor performance while the majority earned high scores. 

Student responses to multiple-choice questions on unit tests were also totaled and 
compared. Performance on each test item was assessed and a “1” was recorded for 
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correct responses and a “0” was recorded for incorrect responses. Each set of questions 
yielded the same total points enabling easy comparison of question sets. 

On the first test, frequency of responses on three questions not related to Piaget, 
three questions on Piaget’s theory at the application level, and three questions on Piaget’s 
theory at the knowledge/comprehension level was determined, total point value for each 
question set was 45. Figure 3 shows the total scores on the three sets of questions from 
the first test. Frequency of correct responses was the highest for knowledge level 
questions about Piaget’s theory. The score for the control set of questions was lower than 
the knowledge level set of questions and only 1 point higher than the total score for the 
application level questions. Frequency of responses to multiple-choice questions was also 
calculated for the test on motivation. Student responses to two questions on topics other 
than motivation, two questions on motivation at the knowledge level, and two questions 
on motivation at the application level were tabulated and Figure 4 shows the results of 
those calculations. Total scores for each set of questions was 30 points. Total points 
earned on the control questions was 23, total points earned on analysis level questions 
was 29, and total points earned by the students on the identification/knowledge level 

Figure 3. Frequency scores for multiple-choice questions on Piaget. N— 15 
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questions was 28. Results indicate that students performed better on the questions related 
to the topic of motivation than they did on the questions about topics not addressed with 
causal influence diagrams. 



Figure 4. Frequency scores for multiple choice questions on motivation. N=15 
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Student performance on the analytical CID and a case analysis on a test were 
compared to determine the relationship between causal influence diagramming activities 
and performance on the case analysis. Table 4 presents the mean, median, mode, and 
standard deviation for both the diagram and the case analysis from the test. Higher modes 
for both measures indicate that most students performed well on the diagram and on the 
case analysis. The decrease in the standard deviation and high mean on the case analysis 
indicate that even more students were successful on that assessment item. 



TABLE 3. Comparison of performance on CID to performance on motivation case 
analysis. N=15 



13 


Median 


4 


Causal Influence Diagram 




Motivation Case Analysis 


(1-15) 




ik 51 


11.87 


Mean 


4.26 


14 


Mode 


4 


3.642 


Standard Deviation 


0.593 



Student responses to questions about their perceived value of causal influence 
diagramming were also collected and analyzed. Student perceptions of how causal 
influence diagramming activities affected their understanding of the complex systems of 
Piaget’s theory, IDEA, and motivation were measured by responses to three questions. 
The students responded anonymously to the following questions: 

1. To what extent did causal influence diagramming affect your understanding of 
Piaget’s theory, IDEA, and motivation theories? 

2. To what extent did causal influence diagramming improve your understanding 
of the relationship between motivation theory and teaching practice? 

3. Did you use Inspiration for other classes? If yes, briefly describe how and why 
you used the software. 

For question 1 , student responses were categorized at three levels, did not affect, affected 
somewhat, and did affect. Figure 5 presents the frequency of student responses to 
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question 1 and reveals that the majority of students believed that causal influence 
diagramming activities had a positive affect on their performance. 

Figure 5. Frequency of responses to open-ended question 1. N=14 



Figure 6 shows the frequency of responses to the second open-ended question. Student 
responses were categorized in three areas, did not improve, improved somewhat, and 
improved. Results of frequency analysis revealed that the majority of students perceived 
the causal influence diagramming activities as helpful. 

Figure 6. Frequency of responses to open-ended question 2. N=14 





Students were also asked if they used the software application for the causal influence 
diagramming activities in other classes. Figure 7 presents the frequency of responses to 
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question 3 and reveals that most students did not use this application to support work in 
other classes. 

Figure 7. Frequency of student responses to question 3, N=14 




It is evident that this group of students perceived their experiences with causal influence 
diagramming as helpful on the study of complex educational systems. Students perceived 
descriptive causal influence diagramming as an effective tool to support understanding of 
complex systems. Fewer students indicated that analytical causal influence diagramming 
helped them recognize the relationships between theories of motivation and classroom 
practice. Slightly less than 1/3 of the students reported that they used the software 
application to help them in another class. 

Discussion 

Most students performed well on the causal influence diagrams and the test items. 
Comparison of student performance on multiple choice test questions related to both 
types of causal influence diagramming activities and multiple choice questions not 
related to the causal influence diagramming reveals that the activities affected student 
performance in a positive way. Student performance on the first test showed that correct 
responses to questions at the knowledge/comprehension level exceeded correct responses 
to the set of control questions. Accuracy on the application/analysis/synthesis level 
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questions almost equaled the accuracy of responses to the set of control questions. 
Additionally, student self-reports show that attitudes toward the activities were positive. 

Comparison of performance on multiple-choice test items on motivation revealed 
that scores for both sets of motivation questions exceeded the total group scores for the 
set of control questions and performance on analysis/application level questions was 
slightly higher than performance on knowledge/comprehension level questions. The 
majority of students felt that the activities were beneficial in helping identify 
relationships between theory and practice, and the performance on the test items supports 
their perceptions. The comparison of performance on the analytical causal influence 
diagram and analysis of a case on a test support the idea that critical analysis skills 
transfer. Students who performed well on the diagram performed well on the test. 
Although causality between performance on diagramming activities and test performance 
cannot be substantiated by this study, it can be assumed that the activities had some 
influence on student performance. The results of multiple-choice test questions on 
motivation theories indicate that the combination of a case study and a causal influence 
diagramming activity affected student performance more than descriptive causal 
influence diagramming. Student experience with the software, the activities, and more 
guided practice complicate interpretation of these results. It appears, however, that the 
combination of a case study and a causal influence diagramming activity had more 
impact on student performance than descriptive diagramming assignments. 

Implications for Teacher Educators 
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The results of this study provide a starting point for a more controlled 
investigation of the effects of causal influence diagramming on the analysis of complex 
educational systems. Since student performance was consistently high on the diagrams 
and the multiple choice test questions on the diagrams, performance at causal influence 
diagramming can be a strong indicator of performance on traditional evaluation 
measures. Student self-reports indicate positive attitudes toward the activities, most likely 
facilitated by unlimited access to a computer and the software throughout the course. 

A case study approach to an analytical causal influence diagramming activity has 
the potential to improve student performance beyond that descriptive diagramming alone. 
Integrating a systems analysis approach and computer software to facilitate the 
representation of complex systems in an educational psychology course is an excellent 
method for improving test performance, helping novice teachers recognize the value of 
non-traditional computer-based approaches to assessment, and modeling methods for 
addressing diverse learning styles. Engaging future teachers in such experiences 
throughout all phases of their preparation can only increase the likelihood that these 
methods will be integrated into their instructional design repertoire in the future. 
Meanwhile, causal influence diagramming activities prepare them to move into a closer 
study of the complex domain of teaching and learning in their professional preparation 
courses. 
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